Moral Judgments of Human vs. AI Agents in Moral Dilemmas

Artificial intelligence has quickly integrated into human society and its moral decision-making has also begun to slowly seep into our lives. The significance of moral judgment research on artificial intelligence behavior is becoming increasingly prominent. The present research aims at examining how people make moral judgments about the behavior of artificial intelligence agents in a trolley dilemma where people are usually driven by controlled cognitive processes, and in a footbridge dilemma where people are usually driven by automatic emotional responses. Through three experiments (n = 626), we found that in the trolley dilemma (Experiment 1), the agent type rather than the actual action influenced people’s moral judgments. Specifically, participants rated AI agents’ behavior as more immoral and deserving of more blame than humans’ behavior. Conversely, in the footbridge dilemma (Experiment 2), the actual action rather than the agent type influenced people’s moral judgments. Specifically, participants rated action (a utilitarian act) as less moral and permissible and more morally wrong and blameworthy than inaction (a deontological act). A mixed-design experiment provided a pattern of results consistent with Experiment 1 and Experiment 2 (Experiment 3). This suggests that in different types of moral dilemmas, people adapt different modes of moral judgment to artificial intelligence, this may be explained by that when people make moral judgments in different types of moral dilemmas, they are engaging different processing systems.


Introduction
Artificial intelligence (AI) technology has developed rapidly in the past few decades and has been widely used in various fields, taking on roles in diagnostic treatment [1], autonomous driving [2], criminal sentencing assessment [3], and wealth management consulting [4]. However, these decisions are closely related to people's property, as well as physical and mental health, and even involve people's life and death. Thus, the application of new technology should not only consider its utility but also be cautious about the social impact it may cause. As a result, numerous studies have discussed the problem of the ethics of artificial intelligence's decisions [5][6][7], focusing on whether the artificial intelligence agents should be responsible for the negative outcomes of decisions they made and how much blame it should bear [8], whether they follow certain biases of their designers, and whether they have intents or motives to commit harmful acts [9,10].
These discussions reflect the fact that people would not perceive AI as mere technological tools used by human agents [11], but sometimes perceived them as moral agents that could act autonomously and be accountable for their behaviors. However, existing research has not reached a consistent conclusion about how people make moral judgments about AI behaviors and decisions. On the one hand, research showed that people tend to make harsher moral judgments of AI agents' behaviors [12], and resist AI agents making moral decisions [13,14]. On the contrary, there was also evidence that people were more tolerant of AI agents compared to human agents when making moral judgments [6]. In response to these contradictory results, the current study aims to compare people's moral judgments of Behav. Sci. 2023, 13, 181 2 of 14 human agents and artificial intelligence agents in moral dilemmas and examine what kind of moral norms people apply to AI agents.

Moral Judgments
As a core concept of moral cognition, moral judgment refers to the evaluative judgments that a perceiver makes in response to a moral norm violation, including four major classes of judgment: evaluations, norm judgments, wrongness judgments, and blame judgments, from simple to complex information processing [15]. Evaluations consist of evaluations of good and bad, positive and negative, and represent one of the most basic human responses [16]; evaluative priming can occur without feelings and usually within 1600 milliseconds [17,18]. Norm judgments consist of whether something is permissible, required, forbidden, and so forth based on peoples' understanding of social rules [15]. Norm judgments are rather different from moral evaluations. Norm judgments invoke the standards against which evaluations are measured and thus set the context for any judgments that are to be called moral [19]. Moral wrongness judgments merge evaluations and norm judgments of intentional actions [15], reflecting an instinctive focus on the negative aspects of events [20,21], such as the tendency of individuals to automatically search for what is wrong with an event [22], and in some cases, people would firmly believe something is wrong even if they could not articulate the reasons for their judgments (moral dumbfounding) [23]. Blame judgments build on all three processes. An initial blame value is hypothesized to be formed from evaluations and wrongness judgments in light of the seriousness of the violated norm [24,25]. Of all moral judgments, blame appears to be the most flexible, complex, and sophisticated, it requires cognitive resources to integrate morally relevant information from multiple sources (e.g., degree of harm, the agent's causal involvement, intentionality, the agent's reasons for acting, and counterfactual preventability) to complete the presumption of the blame of the agent [15].

The Dual-Process Theory of Moral Judgment
When faced with moral dilemmas, people may make one of two contradictory choices: utilitarian (or, more broadly, consequentialist) behavior, characterized by the actor making the decision with the aim to maximize benefits and minimize costs across affected individuals [26]; on the contrary, deontological behavior refers to the actor emphasizes responsibilities, rights, and obligations regardless of the outcome [27]. Take the trolley dilemma as an instance (there are five people tied to the track in front of a speeding trolley, you can operate a switch to redirect the trolley onto a side rail, but there is also one person tied to the side rail, would you operate the switch to redirect the trolley?). The act of operating the switch (action) can realize the result of "killing one to save five", which is a utilitarian act. On the contrary, not operating the switch (inaction) can preserve the life of the "innocent" person on the side rail, that is, never killing to save more lives, which is a deontological act [28]. The two contradictory types of action may elicit completely different moral judgments from the observers.
The dual-process theory of moral judgment explained how the observers' response process influences their moral judgments about the actors' utilitarian act or deontological act [29,30]. This theory holds that people's harsher moral judgments about utilitarian actions are driven by automatic negative emotional responses; while approval of harmful utilitarian actions is driven by controlled cognitive processes [31]. Both automatic emotional responses and more controlled cognitive responses play crucial and, in some cases, mutually competitive roles in individuals' moral judgments; people's moral judgments of an act may depend on which process outcompetes in this conflict [29,30]. The most direct and compelling evidence still comes from studies about moral dilemmas. The utilitarian act in the trolley dilemma (i.e., operate the switch to "kill one to save five") would not evoke the judge's negative emotional response, at least not very strongly, in this case, the judge makes moral judgments driven by controlled cognitive processes and thus considers the utilitarian choice is more acceptable. In the footbridge dilemma (the actor has to choose Behav. Sci. 2023, 13, 181 3 of 14 between allowing five people to die from a speeding trolley or pushing someone off a footbridge to stop the trolley, saving the five people further down the track, but killing the person pushed), the utilitarian act (i.e., push someone off a footbridge to "kill one to save five") elicits a prepotent negative emotional response of observers. It could be because the harm, in that case, is more intentional [32,33], more direct [34], and involves intervention and personal force on the victim [32,35,36], or for some other reason. In this case, the negative emotional response that favors deontological choice conflicts with and typically outcompetes the controlled cognitive processes that favor utilitarian choice, so the observer would judge the action of pushing someone off a footbridge as more morally wrong and unacceptable [29,30].

Artificial Intelligence as Moral Agents
With the advancement of artificial intelligence technology and the popularization of its application in daily life, artificial intelligence has gradually been involved in moral events that only humans participated in in the past [11]. In fact, artificial intelligence agents have been regarded as moral agents to a certain extent because of technological developments and people's perceptions of them. Yagoda and Gillan [37] suggested that technology develops along two dimensions: intelligence and autonomy. Artificial intelligence is capable of autonomously performing various tasks and making decisions without human supervision, and represents the technology of the highest intelligence and autonomy. For example, artificial intelligence can independently complete specific tasks such as recruitment, financial analysis, product recommendation, and medical care [38,39]. Therefore, AI agents are often seen to have some agency [40,41], which refers to the capacities such as communication, planning, and memory [40]. Mind perception is the essence of moral judgment, dimensions of mind perception (agency and experience) map onto moral types (agents and patients), among them, agency, in particular, qualifies entities as moral agents, those who are capable of doing morally good or wrong [42]. In summary, the high intelligence and autonomy of artificial intelligence lead people to perceive AI agents as having mind perception of agency, while agency is linked to moral agents; therefore, AI agents may be regarded as moral agents and thus could be morally judged by people when involved in moral events.
On the other hand, the practical applications of artificial intelligence agents have already sparked a heated debate about their ethics and responsibility as moral agents in social life and academic literature. Take autonomous vehicles (AVs) as a typical example, which are expected to account for 75% of vehicles on the road by 2040 [43]. AVs could increase traffic efficiency and reduce accidents [44,45]; however, not all crashes will be avoided, and some crashes will require AVs to make difficult moral decisions, in cases that involve unavoidable harm-running over pedestrians or sacrificing themselves and its passenger to save them [46]. Car manufacturers and policymakers are currently struggling with these moral dilemmas, in large part because these problems involved a conflict between two moral principles-utilitarianism and deontology. For example, Bonnefon et al. [46] found that people approved of utilitarian AVs (that sacrifice their passengers for the greater good), and would like others to buy them, but they would themselves prefer to ride in AVs that protect their passengers at all costs. If AVs and other autonomous agents do not embed moral principles to guide their decisions in a way that is acceptable to people, there may be some negative outcomes for consumers and manufacturers such as stirring public outrage and discouraging buyers, and thus the world-changing benefits of artificial intelligence agents would also be lost as a result. Accordingly, as we are about to endow countless machines with autonomy, taking AI morality seriously has never been more urgent.

Moral Judgments of Human versus AI Agents
Some studies have explored whether people apply the same moral norms to AI and human agents or not, that is, whether people's moral judgment of human and AI agents would differ when they make a certain choice (a utilitarian or deontological act). Malle et al. [6] and Voiklis et al. [7] found that people may apply moral norms differentially to humans and AI agents: AI agents are expected-and possibly obligated-to make utilitarian choices. Specifically, participants regarded the act of sacrificing one person to save four (a utilitarian choice) as more permissible for a robot than for a human, a robot that chose this sacrifice was considered morally wrong by far fewer people than a human agent who made that same choice, and human agents were blamed considerably more for taking action than for refraining, whereas robots received almost as much blame for refraining as for taking action. Conversely, Komatsu et al. [47] found that neither the types of agents (human or robot) nor the types of actions (action or inaction) affect the participants' judgment about moral wrongness for humans and robot agents, that is, people apply the same moral norm to humans and AI agents. Moreover, Bigman and Gray [8] found that people are averse to machines making moral decisions in dilemmas that directly impact human life and death.

The Current Research
Across three studies, we investigated people's moral judgments of human versus artificial intelligence agents in moral dilemmas. In Experiment 1, participants read about either a human agent or an AI agent who faced a trolley dilemma, and then they read whether the agent's actual act was action (a utilitarian act) or inaction (a deontological act). In Experiment 2, participants read either a human or an AI agent who faced a footbridge dilemma and their actual actions. In Experiment 3, participants read about both a trolley dilemma and a footbridge dilemma where the agent (a human or an AI agent) performed either a utilitarian act or a deontological act. To measure people's moral judgments of humans and AI agents, participants rated the morality, permissibility, wrongness, and blameworthiness of the agent's behavior after reading the scenarios in all three studies (Experiments 1-3).

Experiment 1
The purpose of Experiment 1 is primarily to examine whether people apply the same or different moral norms and judgments to human agents and AI agents in moral dilemmas that people are typically driven by controlled cognitive processes, using the most classic trolley dilemma as an experimental paradigm. Previous research has found that AI agents, compared with humans, were more commonly expected to make a deontological decision, that is, sacrifice one person for the good of many, and they were blamed more than humans when they refrained from that decision in trolley dilemmas [6,7]. However, it has also been found that people apply the same moral norms and judgments to human and AI agents in trolley dilemmas [47]. Thus, we conducted a validation test about moral judgments about human and AI agents' behavior in the trolley dilemma in Experiment 1.

Participants
One hundred and ninety-five undergraduate students (Mage = 19.24, SD = 1.84, 111 females, 84 males) participated in this study for course credits. This experiment was approved by the Institutional Review Board (IRB) of the authors' university and all participants gave informed consent.

Materials
Participants read about either a human agent or an AI agent who faced a trolley dilemma [48]. In the trolley problem scenario, the main character had to choose between allowing five people to die from a speeding trolley (inaction) or operating a switch that redirects the trolley onto a side rail, which will save the five people but kill another person (action). A picture describing the scenario was presented below the text at the same time.

Design and Measures
We randomly assigned participants to a 2 (Agent Type: human vs. AI) × 2 (Action: action vs. inaction) between-subjects design. After consenting, participants completed the experiment through Qualtrics, an online survey software program. We experimentally varied the factor Agent Type by describing the main character as either a "railway worker" or an "artificial intelligence program". We also experimentally varied the factor Action by stating that the agent either did or did not direct the trolley toward the single person. After reading the scenario and learning which action the main character actually chose, participants were asked to indicate the morality, permissibility, wrongness, and blameworthiness of the agent's behavior (rated on a 100-point slider scale; adopted from [6,7]). Lastly, they answered demographic questions including their age and gender.

Results
Morality. A 2 (Agent Type: human vs. AI) × 2 (Action: action vs. inaction) betweensubjects analysis of variance using morality as a dependent measure revealed a significant main effect of agent type, allowing five people to die from a speeding trolley (inaction) or operating a switch that redirects the trolley onto a side rail, which will save the five people but kill another person (action). A picture describing the scenario was presented below the text at the same time.

Design and Measures
We randomly assigned participants to a 2 (Agent Type: human vs. AI) × 2 (Action: action vs. inaction) between-subjects design. After consenting, participants completed the experiment through Qualtrics, an online survey software program. We experimentally varied the factor Agent Type by describing the main character as either a "railway worker" or an "artificial intelligence program". We also experimentally varied the factor Action by stating that the agent either did or did not direct the trolley toward the single person. After reading the scenario and learning which action the main character actually chose, participants were asked to indicate the morality, permissibility, wrongness, and blameworthiness of the agent's behavior (rated on a 100-point slider scale; adopted from [6,7]). Lastly, they answered demographic questions including their age and gender.

Results
Morality. A 2 (Agent Type: human vs. AI) × 2 (Action: action vs. inaction) betweensubjects analysis of variance using morality as a dependent measure revealed a significant main effect of agent type,  All the results of Experiment 1 are summarized in Table 1.

Experiment 2
In Experiment 1, we found that in the trolley dilemma, people seem to be less concerned about whether the main character in the dilemma makes a utilitarian choice or a deontological choice, but focus more on whether the decision maker is a human or an artificial intelligence agent, people are averse to AI making moral decisions. However, most of the previous studies on moral judgments about AI have been conducted in the context of the trolley dilemmas, but fewer in another classic type of moral dilemma, the footbridge dilemma. Thus, in Experiment 2, we examined how people make moral judgments about human and AI agents' behavior in the footbridge dilemma.

Participants
One hundred and ninety-four undergraduate students (Mage = 19.18, SD = 2.57, 115 females, 79 males) participated in this study for course credits. This experiment was approved by the Institutional Review Board (IRB) of the authors' university and all participants gave informed consent.

Materials
Participants read about either a human agent or an AI agent who faced a footbridge dilemma [49]. In the footbridge problem scenario, the main character had to choose between allowing five people to die from a speeding trolley (inaction) or pushing someone off a footbridge and onto the path of that speeding trolley, saving the five people further down the track, but killing the person pushed (action). A picture describing the scenario was presented below the text at the same time. All the results of Experiment 1 are summarized in Table 1.

Experiment 2
In Experiment 1, we found that in the trolley dilemma, people seem to be less concerned about whether the main character in the dilemma makes a utilitarian choice or a deontological choice, but focus more on whether the decision maker is a human or an artificial intelligence agent, people are averse to AI making moral decisions. However, most of the previous studies on moral judgments about AI have been conducted in the context of the trolley dilemmas, but fewer in another classic type of moral dilemma, the footbridge dilemma. Thus, in Experiment 2, we examined how people make moral judgments about human and AI agents' behavior in the footbridge dilemma.

Participants
One hundred and ninety-four undergraduate students (Mage = 19.18, SD = 2.57, 115 females, 79 males) participated in this study for course credits. This experiment was approved by the Institutional Review Board (IRB) of the authors' university and all participants gave informed consent.

Materials
Participants read about either a human agent or an AI agent who faced a footbridge dilemma [49]. In the footbridge problem scenario, the main character had to choose between allowing five people to die from a speeding trolley (inaction) or pushing someone off a footbridge and onto the path of that speeding trolley, saving the five people further down the track, but killing the person pushed (action). A picture describing the scenario was presented below the text at the same time.

Design and Measures
The procedure of Experiment 2 was very similar to Experiment 1, with only one difference: the moral dilemma was a footbridge problem above instead of a trolley problem. The manipulation of Agent type was identical to that in Experiment 1. We varied the Action by stating that the agent either did or did not push someone off the footbridge onto the path of that speeding trolley. After reading the scenario and learning which action the main character actually chose, participants responded to the same measures of moral judgments and demographic questions as in Experiment 1.

Results
Morality. A 2 (Agent Type: human vs. AI) × 2 (Action: action vs. inaction) betweensubjects analysis of variance using morality as a dependent measure revealed a significant main effect of action,  Table 2.

Experiment 3
In Experiments 1 and 2, we found that in the trolley dilemma people were more concerned with who made the decision and less concerned with whether it was a utilitarian or deontological decision; more specifically, people were averse to AI making moral decisions about human life and death. In contrast, in the footbridge dilemma, people pay less attention to whether the decision maker is a human or an artificial intelligence agent but pay more attention to whether the agent makes a utilitarian decision or a deontological decision; more specifically, people were averse to a utilitarian decision in the footbridge dilemma regardless of whether the decision maker is a human or an AI agent. To verify the consistency of the results of Experiments 1 and 2, and to further explore the differences in people's moral judgments about human and AI agents' behavior in different types of dilemmas, we conducted a within-subject experiment in Experiment 3, using Dilemma Type as a within-subject factor.

Participants
Two hundred and thirty-six undergraduate students (Mage = 18.93, SD = 1.67, 147 females, 89 males) participated in this study for course credits. This experiment was approved by the Institutional Review Board (IRB) of the authors' university and all participants gave informed consent.

Materials
The moral dilemma scenarios were identical to those in Experiments 1 and 2.

Design and Measures
The experiment utilized a 2 (Dilemma Type: trolley dilemma vs. footbridge dilemma) × 2 (Agent Type: human vs. AI) × 2 (Action: action vs. inaction) mixed design with Dilemma Type as a within-subject factor and both Agent Type and Action as betweensubject factors. We randomly assigned participants to one of each condition. After consenting, participants completed the experiment through Qualtrics, an online survey software program. Participants read both two scenarios, with half the sample reading the trolley dilemma first and another half the sample reading the footbridge dilemma first. The manipulations of Agent Type and Action were identical to those in Experiments 1 and 2. In the end, participants responded to the same measures of moral judgments and demographic questions as in Experiments 1 and 2.

Discussion
In the trolley dilemma (Experiment 1), the agent type rather than the actual action influenced people's moral judgments. Specifically, participants rated AI agents' behavior as more immoral and deserving of more blame than humans' behavior, regardless of whether they act utilitarianly or deontologically. Conversely, in the footbridge dilemma (Experiment 2), the actual action rather than the agent type influenced people's moral judgments. Specifically, participants rated action (a utilitarian act) as less moral and permissible and more morally wrong and blameworthy than inaction (a deontological act), regardless of whether the actor is a human or an AI agent. The result of Experiment 3 provided a converging pattern that, in the trolley dilemma, agent type influenced people's moral judgments: participants rated human agents' behavior as more permissible than AI agents' behavior. On the contrary, in the footbridge dilemma, only action influenced people's moral judgments, participants rated action as less moral and permissible and more morally wrong and blameworthy than inaction; agent type did not influence people's moral judgments in this dilemma. There was one small difference between Experiment 3 and Experiment 1. In Experiment 1, only agent type influenced people's moral judgments, while in Experiment 3, people were interested in both the difference between humans and AI and the action versus inaction; people rated the action as less moral and more morally wrong than inaction in the trolley dilemma. It may be explained that in Experiment 3, people read both the two types of moral dilemmas and people's focus may be influenced by the previous scenario when making moral judgments about the second scenario.
Overall, these findings revealed that, in the trolley dilemma, people are more interested in the difference between humans and AI agents than action versus inaction. Conversely, in the footbridge dilemma, people are more interested in action versus inaction. It may be explained that people made moral judgments driven by different response processes in these two dilemmas-controlled cognitive processes occur often in response to dilemmas such as the trolley dilemma and automatic emotional responses occur often in response to dilemmas such as the footbridge dilemma [30]. Thus, in the trolley dilemma, controlled cognitive processes may drive people's attention to the agent type and make the judgment that it is inappropriate for AI agents to make moral decisions. In the footbridge dilemma, the action of pushing someone off a footbridge may evoke a stronger negative emotion than the action of operating a switch in the trolley dilemma. Driven by these automatic negative emotional responses, people would focus more on whether the agents did this harmful act, and judged this harmful act less acceptable and more morally wrong.
However, it should be noted that our work presents some limitations and offers several avenues for future research. Firstly, the current study only examined how people make moral judgments about humans and AI agents, but did not investigate the underlying psychological mechanism. Thus, all interpretations of the results are speculations. Future research could further explore the reason why people are reluctant to AI agents making moral decisions in the trolley dilemma, why people apply the same moral norms to humans and AI agents in the footbridge dilemma, and why people show different patterns of moral judgment in the trolley dilemma and the footbridge dilemma. Previous research has provided us with some pointers. For example, interpretability and consistency of behaviors would increase people's acceptance of AI [50,51]; increased anthropomorphism of an autonomous agent would mitigate blame for an agent's involvement in an undesirable outcome [52]. Individual differences including personality [53,54], development experiences [55][56][57], and cultural background [58] may also influence people's attitudes toward AI agents. Second, to exclude the potential influence of individual differences between Experiments 1 and 2, we conducted Experiment 3 with a within-subjects design, participants were asked to read both the two scenarios; however, the processing system activated by the first scenario may influence the participants' judgment about the subsequent scenario. For example, participants who read the footbridge dilemma first may be interested in whether the character acted or not due to the strong negative emotion, this emotion may drive people to focus on the character's action in the subsequent trolley dilemma, just like they did in the footbridge dilemma. Future research could consider other method approaches to exclude the effects of individual differences and order effects.
Last but not least, we examined people's moral judgments of humans and AI agents using the most classical and traditional dilemma paradigms. However, recent studies have discussed the method limitations of these traditional dilemma paradigms. For example, Gawronski et al. [59] claimed that the conceptual meaning of responses in the traditional paradigm is ambiguous because the central aspects of utilitarianism and deontologyconsequences and norms-are not manipulated. This shortcoming may undermine empirical findings such as the problem that the traditional moral paradigms could not measure the general action/inaction biases of participants. Given this, there might be other explanations for the current results, such as the inclination to be overall acceptable to any behavioral proposals. Specifically, there might be a stronger overall inclination to accept the action choices in the trolley dilemma than in the footbridge dilemma. This is an alternative explanation that should be tested in future studies. To resolve these limitations of traditional dilemma paradigms, Gawronski et al. presented a multinomial model (the CNI model) that allows researchers to quantify sensitivity to consequences, sensitivity to moral norms, and a general preference for inaction versus action irrespective of consequences and norms in responses to moral dilemmas (for details, see [59,60]). More recently, however, the limitations of the CNI model were also been discussed (for details, see [61,62]). As a result, Liu and Liao [63,64] developed a new algorithm-the CAN-to address the methodological limitations of the CNI model and fix these limitations (for details, see [63][64][65]). We suggest that further research could examine people's moral judgments of humans and AI agents using these newly developed approaches and make a comparison with the current results that come from the traditional dilemma paradigms.